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An Unobtrusive Experimental Evaluation oj 
Systems Approach for Teaching Reading 

by 

Carl D. Novak 
Educational Service Unit No. 18 
Lincoln, Nebraska 



This study was conducted as part of the ongoing evaluation of 
Project INSTRUCT, a reading project hosted by the Lincoln (NE) Public 
Schools and funded by Title III, ESEA. 
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Project INSTRUCT is an acronym for Instructional System for Teaching 
Reading Using Continuous Progress Technology. The system is based on an 
array of essential skills in word attack and reading. Mastery tests are 
used to place students on the array and to monitor progress. Instruction 
is focused through the grouping and regrouping of students so that they 
are always working on an appropriate skill building activity. 

The achievement oriented goals of Project INSTRUCT are (1) to im- 
prove reading achievement and (2) prevent reading failure among primary 
students. This study focused on the effect of participation in Project 
INSTRUCT on achievement measured by standardized test scores. 



Perspective 

Two studies conducted during the 1971-72 school year indicated that 
primary students in Project INSTRUCT schools tended to score higher on the 
reading subtests of nationally normed tests than students in schools not 
using Project INSTRUCT. Both studies had technical and/or theoretical 
shortcomings. The first study was a comparison of the stanine distribution 
in Project INSTRUCT and non-Project INSTRUCT schools. The second was an 
analysis of covariance study of student achievement. The second study was 
based on the performance of 90 students from six Project INSTRUCT schools 
and 90 students from non-Project INSTRUCT schools. Implementation of the 
Project INSTRUCT system was not taken into account. 

rH 

The results of the first study indicated that the proportion of stu- 
dents that received high stanine scores (7 - 9) in reading and reading 

O related tests was greater in schools using Project INSTRUCT than in similar 
schools not using Project INSTRUCT. The proportion of students that received 
average scores (4 - 6) also was greater. Consequently a smaller proportion 
. of students in Project INSTRUCT schools received low (1 - 3) stanine scores. f 

There were no differences between stanine distributions in Project INSTRUCT 



^Presented at the Annual Meeting of American Educational Research 
Association, Washington, D.C., March, 1975. 



schools and non-Project INSTRUCT schools prior to the implementation of 
Project INSTRUCT. The results of the second study indicated that students 
in Project INSTRUCT consistently scored higher than comparable control groups 
on the reading related subtests of a nationally normed achievement test, the 
Metropolitan Achieverent Test. The differences, however, were not statisti- 
cally significant. 

The purpose of this study was to confirm the findings of these two 
earlier studies and to control for the additional factor of degree of imple- 
mentation of Project INSTRUCT. 



flGthodology 

The basic analytical procedure v/as analysis of covariance. Three 
separate analyses were conducted utilizing three criterion variables. The 
three variables were the Word Knowledge, Reading, and Spelling subtests of 
the Fall 1973 administration of the Primary II Metropolitan Achievement 
Test . The raw scores on the Word Knowledge and Reading subtests of the 
FaTT 1972 administration of the Primary I Metropolitan Achievement Test 
and percentile scores from the Fall 1972 administration of the Otis Lennon 
Mental Ability Test were used as covariants. All tests were administered 
as part of the normal district wide fall testing program. 

Although Project INSTRUCT was developed for use with students in 
grades one through three, the study was restricted to students in the 
second grade. The achievement and ability score-- of a random sample of 
330 students, 165 from Project INSTRUCT schools and 165 from non-Project 
INSTRUCT schools, were analyzed. The sample of Project INSTRUCT students 
was selected from a pool of second grade students who had attended schools 
that had successfully implemented Project INSTRUCT. The criterion for 
successful implementation of Project INSTRUCT by a school was a score of 
300 points or higher on a 550 point implementation scale. Tv/elve of the 
30 elementary schools in the district participated in Project INSTRUCT 
during the 1972-73 school year. Eleven of the 12 received implementation 
scores that exceeded the 300 point minimum. The eight Project INSTRUCT 
schools selected to participate in the evaluation included eight of the nine 
schools that achieved a score of 350 points or higher on the implementation 
scale. Marginal schools, schools with implementation scores greater than 
300 but less than 350 were eliminated from consideration. The ninth school 
that surpassed the 350 point cutoff was eliminated from the study because 
it had been used by Project INSTRUCT as a developmental school. The imple- 
mentation scores of those eight schools ranged from 361 to 476. Each of the 
eight Project INSTRUCT schools was then paired with a non-Project INSTRUCT 
school. The pairs were matched on (1) general socio-economic climate, 
(2) organizational pattern, multi-unit or conventional, and (3) compensatory 
education target school status. Equal numbers of students were then randomly 
selected from the paired Project INSTRUCT and non-Project INSTRUCT schools. 
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Results and Conclusions 

The adjusted raw scores for the Project INSTRUCT and non-Project INSTRUCT 
students are presented in Table I. 

Significant differences favoring the Project INSTRUCT students were 
found on two of the three subtests. The difference on the Word Knowledge 



TABLE I 

Adjusted Raw Scores on the Primary II 
Metropolitan Achievement Test 



Project 

INSTRUCT Control Difference 



Wor-^ Knowledge 
Reading 
Spell ing 



33.85 
35.30 
22.42 



32.25 
33.84 
21.68 



1 . 60** 
1.46* 
.74 



**Significant at the .01 level of confidence. 
*Significant at the .05 level of confidence. 



subtests was significant at the .01 level of confidence, while the difference 
on the Reading subtest was significant at the .05 level of confidence. The 
difference between the mean scores on the spelling subtest was not signifi- 
cant; however, spelling was not formally taught as part of Project INSTRUCT 
during the 1972-73 school year. 
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Discussion 

The results of this study are consistent with the results of the 
1971-72 achievement studies. The Project INSTRUCT students scored higher 
on the Word Knowledge, Reading, and Spelling subtests. The differences 
between the Project INSTRUCT and control samples were larger the second year. 
The more favorable results are not surprising since (1) the first year was 
a developmental year, and not all the materials necessary to fully implement 
Project INSTRUCT were available for the entire year; (2) neither the Pi'oject 
INSTRUCT staff nor the schools had experience in implementing Project INSTRUCT 
during the first year; and (3) the 1972-73 study was limited to students from 
schools that were the most successful in implementing Project INSTRUCT. During 
the first year of the project only a few schools actually tried to implement 
Project INSTRUCT; therefore, it was impractical to try to exclude schools who 
had trouble implementing Project INSTRUCT from the study of achievement. 

The second year study was also consistent with the first year study 
In that the differences between the Project INSTRUCT and control groups on 
tfie Spelling subtest were smaller than the differences on the other subtests. 
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During these two years Project INSTRUCT did not include a formal spelling 
program; the project dealt exclusively with the word attack and reading 
skills. The results of both achievement studies reflect this emphasis. The 
difference on the Spelling subtest was the only one that was not statisti- 
cally significant. This pattern supports the thesis that Project INSTRUCT 
affects reading achievement. 



Conclusion 

The results of the 1972-73 evaluation of student achievement indicate 
that Project INSTRUCT does influence student achievement in a positive 
direction. The scores of students from Project INSTRUCT schools were 
significantly higher than the scores of a comparable sample of students 
from control schools on the Word Knowledge and Reading subtests. These 
findings were consistent with the findings of an earlier study of student 
achievement and with an analysis of the score distributions in Project 
INSTRUCT and non-Project INSTRUCT schools. 



Significance 

The study >s an example of an unobtrusive evaluation study based on 
a traditional research paradigm that was conducted in a public school 
setting. Achievement and ability scores were sampled at the central office 
without involving teachers and/or administrators. Implementation scale 
scores that were used to identify schools that had satisfactorily implemented 
the program were already available from a previous evaluation study. 
Results indicate that the program was a success. 



